DOCUMENT RESUME 



ED 362 398 



SE 053 671 



AUTHOR 
TITLE 

PUB DATE 
NOTE 



PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



ABSTRACT 



Gottfried, Sandra S. 

A Formative Evaluation of a Scientific Work 
Experience Program for Science Teachers. 
Apr 93 

30p.; Paper presented at the Annual Meeting of the 
National Association for Research in Science Teaching 
(Atlanta, GA, April 15-19, 1993). 
Reports - Research/Technical (143) — 
Speeches/Conference Papers (150) -- Tests/Evaluation 
Instruments (160) 

MF01/PC02 Plus Postage. 

"College School Cooperation; Experiential Learning; 
Formative Evaluation; Higher Educu-cion; Inservice 
Teacher Education; '^Internship Programs; 
'^Researchers; Science Instruction; -'Science Teachers; 
'•Scientific Research; Secondary Education; Secondary 
School Science; Secondary School Teachers; '''Teacher 
Attitudes; Teacher Behavior; Teacher Education 
Programs; Work Experience Programs 
'•Science Teachers as Research Scientists Program; 
University of Missouri Saint Louis 



The Science Teachers as Research Scientists (STARS) 
program IS a scientific work experience program conducted at the 
University of Missouri-St. Louis that provides secondary science 

Jk^JtIrq'"''^'""'^'^' ""^^ ^^^""^^y researchers. This report evaluates 
the STARS program with respect to the teachers' (n=17) first summer 
experiences only. Both quantitative and qualitative measures were 
used to evaluate the program's effectiveness and to determine whether 
the project objectives were met, the value of the program as 
perceived by the participants, and the impact of the program. The 
study concluded that teachers perceived that: (1) the experience in 
scientific research design and experimentation was substantial; (2) 
they made gains in their acquisition of science content knowledge; 
(3) they moderately increased their ability to perform science 
process skills; (4) they moderately increased the incorporation of 
research/process methodologies in their curricula; (5) they made 
^ho'^''"^f^ understanding of science applications in 

the workplace; (6) the program was helpful in lesson planning; (7) 
the program was valuable; and (8) the program impacted their ability 
to implement hands-on laboratory-oriented curricula. Field notes and 
classroom observations, however, do not reflect an increase in the 
use of these strategies in the classroom, suggesting the need for 
further research concerning how these changes in perception can be 
converted to changes in the classroom. (MDH) 



« Reproductions supplied by EDRS are the best that can be made * 
j,^^^..... from the original document. * 



A FORMATIVE EVALUATION OP 
A SCIENTIFIC WORK EXPERIENCE PROGRAM FOR SCIENCE TEACHERS 



Sandra S, Gottfried 
Departments of Biology and Educational Studies 
University of Missouri-St, Louis 
St. Louis, MO 63121-4499 



Paper Presented at the Annual Meeting of the 
National Association for Research in Science Teaching 

Atlanta , Georgia 

April, 1993 



U.S. DCPARTMENT Of EDUCATION 
OfftCff o< Educational Re»earch and improvement 

EDUCATfONAL RESOURCES INFORMATION 
CENTER (ERIC) 

QCThts documeni hta b«en reprcxlucftd as 
received from the p«raon or organization 
originating it 

□ Minor Changes have been made to improve 
reproductiOT) quality 



Poi nt I of view or opinior^i stated in this docu- 
ment do not necetaanly represent oHtctai 
OERI potition or pohcy 



"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 

Sandra S. Gottfried 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)." 



A FORMATIVE EVALUATION OP 
A SCIENTIFIC WORK EXPERIENCE PROGRAM FOR SCIENCE TEACHERS 

Introduction 

The Science Teachers as Research Scientists (STARS) 
program is a scientific work experience program conducted at the 
University of Missouri-St. Louis (UMSL) . Scientific work 
experience programs are those in which teachers work within 
businesses, industries, or universities in a variety of roles. 
These programs can be grouped into three categories according to 
the type of scientific work teachers do: project internships, 
research internships, and a combination of both project and 
research internships. In project"^ internship programs, teachers 
are placed within business or industry and complete a project for 
the company. In research internship programs, teachers are placed 
in business, industry, or university settings and work with 
research scientists or engineers. In project/ research internship 
programs, some teachers are placed in business, industry, or 
university research settings, while others are placed in business 
or industry with a project focus. 

STARS is a research program that provides internships with 
faculty researchers on the UMSL campus to secondary level 
teachers in the St. Louis, Missouri metropolitan area. This 
program was developed during the 1989-90 academic year as an 
outgrowth of a 1988 planning session on inservice training of 
teachers. Participants at this meeting, which included members of 
the St. Louis Public School District and the University of 
Missouri-St. Louis faculty, identified factors important in 



science teacher inservice education that could lead to 
improvements in student achievement* These factors include: 
aO enhancement of the content knowledge background of teachers, 
bO increased contact with workplace applications for science 
skills, 

cO a strong understanding and practice of hands-on and process- 
oriented teaching techniques, and 

dO interaction with university faculty to develop needed skills 
and understandings. 

The STARS summer research internship/ curriculum development 

program was developed to meet inservice needs while addressing 

these factors* 

It is important to note that over 80 scientific work 
experience programs are now in existence in the United States 
(Industry Initiatives for Science and Math Education [IISME], 
1991) . Therefore, the implications of the evaluation 
methodologies selected and the results of the evaluation of the 
STARS program has the potential to impact the implementation and 
evaluation of programs across the United States. Currently, the 
primary means of evaluation of scientific work experience 
programs are teacher and mentor qiaestionnaires', teacher 
interviews, and pre-to-post program teacher and student attitude 
questionnaires (Gottfried, et al, 1993) • 



Objectives of the STARS program 



The goal of the STARS program is to provide meaningful 
inservice experiences in scientific research, science' education 
theory, and curriculum development for teachers in the St. Louis 
metropolitan area. The specific objectives of this program are 
to: 

(a) provide teachers with experience in scientific research 
design and experimentation, 

(b) enhance teachers' understanding of the nature of science, 

(c) upgrade teachers' science content knowledge and science 
process skills, 

(d) increase teachers' knowledge regarding applications of 
science in the workplace, 

(e) upgrade teachers' skills in the implementation of learning 
cycle and inquiry strategies in science^ teaching, and 

(f) guide teachers in the preparation of laboratory-based 
curriculum projects based on their research experiences. 

Structure of the STARS program 

The STARS program has been conducted on the University of 
Missouri-St. Louis campus beginning with the selection of the 
first teacher/ interns in the spring of 1990. Eight teachers were 
selected for the 1990 summer program; seven teachers were 
selected for the 1991 summer program, including two "returning" 

3 



ERLC 



5 



teachers from the 1990 program; and five teachers were selected 
for the 1992 suitimer program, including one returning teacher from 
the 1991 program. This report evaluates the STARS program with 
respect to the teachers' first summer experiences only (N-17) • 

Teachers were selected for internships based on written 
applications and interviews that determined a "best fit" of 
research interests among the teachers and the researchers • 
Teachers worked with researchers in the departments of biology, 
chemistry, and physics for six weeks, five days per week during 
the months of June and July. Each teacher's experience was unique 
in that it was determined by the^nature of the research being 
pursued by the faculty scientist with whom s/he worked • In 
addition, the manner and extent of the involvement of the teacher 
in the research was determined by the teacher's ability and 
background, and the ability of the researcher to train the 
teacher in appropriate research methodologies within the time 
available • 

Each Friday morning the teacher/ interns participated in 
curriculum development workshops. The first two sessions of 
these workshops focused on an introduction, or for some a 
review, of learning cycle strategy to develop laboratory 
materials in their classrooms. This teaching strategy was 
selected because it reflects the investigative nature of science. 

After two sessions of introduction, review and application 
of the learning cycle strategy in curriculum development, 
teachers presented a summary of their research projects, and 
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worked to brainstorm ideas for curricular materials based on 
their research. (This was the hardest link for teachers to 
make, although some were quite successful.) During the next two 
workshops (and on their own time) , the teachers developed 
laboratory-based curricular materials to implement in their 
classrooms. During the final session, the teachers presented 
the materials they prepared. In addition to the development of 
curricular materials for their classes, the teachers were also 
instructed in the use of scenarios in their classrooms to help 
students develop expertise in science process skills, such as 
identifying problems, developing-^testable hypotheses, 
identifying independent and dependent variables, summarizing 
data, and drawing conclusions from data. 

After returning to school in the fall and using their 
curricular materials with their students, the teachers further 
refined them. After final revision, they were printed for 
dissemination to colleagues. 

The Formative Evaluation 

Prior to the 1990 summer program, a formative evaluation 
procedure was designed. Both quantitative and qualitative 
measures were used to evaluate the program's effectiveness and 
determine whether project objectives had been met. Along with 
determining whether project objectives had been achieved, other 
critical aspects of the program were also evaluated such as: the 
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value of the program as perceived by the teachers and the 
researchers, the impact of the program on the ability of the 
teacher to implement a hands-on laboratory-oriented curriculum, 
and vhe manner in which the program was structured. Initially, 
five different instruments were used: 

(a) The Test of Integrated Process Skills (TIPS) , developed and 
validated by James Okey of the University of Georgia and F. 
Gerald Dillashaw of Bradley University; 

(b) The Sr: lence Classroom Activity Checklist , developed and 
validated by L.H. Kockendorfer of Valparaiso University; 

(c) two teacher questionnaires developed by the project director; 
and 

(d) one researcher questionnaire developed by the project 
director - 

The results from the researcher questionnaire focus on management 
issues and are not reported here. 

Prior to the beginning of the second summer, a questionnaire 
was designed to gather pre- and post-program data regarding the 
amount of class time teacher/ interns spent using specific 
teaching strategies. These data are reported under the heading 
"Pre- and Post-program Teaching Strategies." In addition, a 
graduate student observed a class of each teacher twice prior to 
the program and twice six months after the program ended for the 
1991 and 1992 programs. The classes visited were the same classes 
that were surveyed using the Science Classroom Activity 
Checklist. Funding and time restrictions precluded the 
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continuance of observations until data saturation was reached. 
The field notes from these observations are currently being coded 
and analyzed. 

The Test of Integrated Process Skills (TIPS) 

The TIPS test was administered to teachers during the 
orientation session and during the final science education 
workshop. The science process skills tested in this instrument 
are those associated with planning, conducting, and 
interpreting results from investigations. Usually referred to 
as the integrated science processes, they include formulating 
hypotheses, operationally defining, controlling and 
manipulating variables, planning investigations, and 
interpreting data. The test consists of 36 items. 

Table 1 lists the means of the pre- and post-test scores the 
teachers obtained on the TIPS test. The mean gain of 1.8 points 
was not statistically significant. However, eight out of the 17 
teachers scored above 80% on the pre-test (29 items correct) , 
suggesting a possible "ceiling effect": these teachers could show 
improvement in the post-test on seven or fewer items. Therefore, 
the TIPS test may not be a rigorous enough evaluation tool for 
this group. The original search for instruments measuring process 
skill development was quite extensive and yielded few 
instruments. The one chosen was considered the most appropriate 
to measure process skill attainment associated with hands-on 
laboratory research. 
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The Science Classroom Activity Checklist 

The Science Classroom Activity Checklist is a 53-item 
instrument in which a student answers true or false to a 
descriptor regarding behaviors of the teacher, behaviors of the 
students, or the nature of the use of textbooks, tests, and 
other materials in the classroom. If the statement describes 
what actually occurs in the classroom, the student response is 
"true." If the statement does not describe what occurs in the 
classroom, the student response is "false." For each statement, 
there is a favored response. For example, item nine reads: "My 
job is to copy down and memorize what the teacher tells us." 
The favored response to this item is "false." 

There are seven categories of items in the Science 
Classroom Activity Checklist. The category names are listed on 
Table 2 . The first eight questions regard the role of the teacher 
in the classroom. Favored responses describe the teacher as 
encouraging student discussion, showing students the tentative 
nature of science, and encouraging students to think beyond facts 
to the development of ideas. The favored responses of the next 
eight questions describe a classroom in which students are the 
center of activity as they probe, question, and ]earn. The 
favored responses of items 17-23 describe textbooks as only one 
source of information used in the classroom, and not as an 
encyclopedia of terms to be memorized. In the category "design 
and use of tests," favored responses indicate that test questions 
are used to evaluate a student's ability to apply information 
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learned in the laboratory, develop answers to new problems, and 
synthesize information from various sources, rather than solely 
focusing on recall. The last three categories focus on the 
laboratory itself. The favored responses to questions- 30-37 
describe lab activities that link to other classroom activities, 
discussions, or questions and are used to investigate phenomena 
rather than to verify statements made by the teacher or the 
textbook. The favored responses to questions 38 - 46 describe 
student-centered, teacher-involved laboratory activities that may 
yield varying data that need not be interpreted in only one 
"correct" way. The favox-ed responses in the last category, lab 
follow-up activities, describe a situation in which students 
compare data with one another, attempt to explain unusual data, 
and are allowed to go beyond the regular laboratory exercise and 
do some experimenting on their own. 

The Classroom Activity Checklist was administered to one of 
each teacher's "general" classes prior to the end of the school 
year. After the STARS program and five months into the 
following school year, the Checklist was administered to an 
equivalent class that met at a similar time of day. The 
pre-STAI:S checklist scores were each determined by totalling the 
number of favored responses and totalling the number of unfavored 
responses in each category for each teacher. The number of 
unfavored responses was subtracted from the favored responses to 
yield a score. However, these scores cannot be compared among 
teachers since each teacher had a different number of student 



respondents, and not all categories contained the same number of 
items. Therefore, the scores were transformed by dividing the 
score by the number of items times the number of students to 
yield equivalent data. T tests were performed to compare the pre- 
and post "-test scores in each category. None of the results were 
statistically significant. 
Suiumer Teacher Questionnaire 

A teacher questionnaire was developed to determine the 
teachers' perceptions regarding the increase in their 
understandings and knowledge in content and pedagogy. This 
questionnaire was administered, along with the TIPS test, on 
the last day of the assistantship. Figure 1 shows that the mean 
scores on a Likert-^type scale ranging from a low of 1 to a high 
of 5 ranged from 3.2 to 3.9. 

Teachers also had an opportunity to comment on each 
question they rated. Comments that appeared frequently were: 

* an expression of excitement regarding working in the 
laboratory, including the idea that the experience impacted 
their understanding of the nature of scientific research; 

* delight in the freedom to visit other labs and talk to 
faculty members, graduate students, and post docs; and 

* the desire to extend the experience to another summer or to 
Saturdays during the year. 

Comments include : 

"By talking with the researchers, their assistants and the 
graduate students, it was like being exposed to tailor-made PBS 
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specials on a daily basis." 

"Hopefully I will be able to convey to students that research 
is not a 30-ininute period of time spent in a lab one day a week 
followed by checks mailed to your home and announcement of 
Nobel Prizes, Research is hard work that takes a long time." 
Six-Month Follov-up Teacher Questionnaire 

A second teacher questionnaire was developed and 
administered to the STARS teachers six months after the summer 
experience, to evaluate the teachers' perceptions of the impact 
of the program on their teaching. Figure 2 shows the first 
portion of their responses. The mean scores on a Likert-type ^ 
scale ranging from a low of 1 to a high of 5 ranged from 3.5 to 
4.1. Representative selected comments teachers made were: 

* "The hands-on experiences reminded me how much can be learned 
by having to think through process." 

* "At my school, students spend two hours weekly in the lab. I 
have developed many new labs, and want to say that my students 
get quality hands-on laboratory, influenced by the STARS 
program. " 

* "The people with whom I worked were inspiring. Their ideas and 

support gave me more confidence in the lab. 

* "I told all of my fellow teachers about the wonderful 
program, as well as the other departments at one whole 
district. " 

* "The experience was positive, and I ended up sharing with 
everyone no matter what subject they taught. I also presented 
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material at the department head meeting." 

* "I am not so afraid of physical science anymore." 

* ''I have incorporated new content into society and technology 
topics from the discipline of chemistry." 

* "I now help seniors with their science projects." 

* "The learning cycle strategy gives me more time to help 
students help themselves." 

* "The learning cycle strategy reminds me that what kids really 
master is important." 

* "I am in the process of developing a packet for the needs of my 
building on the research process for the department to use." 

* "I have curtailed some of my abstract teaching activities and 
gone more toward an investigative approach." 

* "My use of lab work has increased exponentially. I do have to 
rewrite most of the labs, though, to make them more in keeping 
with the strategies learned this past suxamer." 

Figure 3 shows the second portion of the teat:her responses 
from the six-month follow-up questionnaire. These items were 
designed to reflect the categories on the Science Classroom 
Activity Checklist, with the last three categories of the 
Checklist incorporated into one item. The mean scores on a 
Likert-type scale ranging from a low of 1 to a high of 5 show 
that teachers felt the greatest impact of the program regarded 
their use of lab follow-up activities. Representative selected 
teacher comments include: 

* "I now let the students do more discovery — I don't just tell 
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them the answers* I give them an opportunity to find out for 
themselves. " 

* "It reminded me to think process and understanding and not to 
be a slave to the curriculum guide (teach dry facts only)." 

* "The lab is a big change. My students are in the lab 25% of the 

class time, which amounts to 2 days/week every two to three 
weeks. (Some of the students have never done lab work 
before! ) " 

When asked to make suggestions to improve the program, the 
teachers indicated a desire to spend two mornings per week in the 
curriculum development workshop. In addition, they expressed a 
desire to return the following summer to continue their research 
projects. 

Pre- and Post-proareun Teaching Strategies 

In answering post-program questionnaires after the 1990 
program, teachers reported that they perceived a change in the 
amount and kind of teaching strategies toward a greater use of 
hands-on laboratory-oriented strategies as a result of 
participating in the STARS program. To gather further data, the 
teachers in the 1991 and 1992 programs (N=8) responded to a 
questionnaire prior to the start of the summer program and six 
months after the program end. Teachers were given a list of the 
following teaching strategies and told to report the percent of 
class time they spent using each: lecture, laboratory activities, 
non-laboratory hands-on activities, discussion, demonstrations, 
student presentations, reading from the textbook, students 
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answering questions orally, students answering questions from the 
textbook or on worksheets, and other (name or describe) . 

To analyze these data, the Sign Test was used. The sign Test 
is a non-parametric paired-sample test. To analyze the data, 
first two categories of activities were developed from the above 
list: (1) laboratory and/or hands-on activities and (2) non- 
laboratory and/or non-hands-on activities. The teaching 
strategies from the teacher list that were considered as part of 
group (1) are laboratory activities, non-laboratory hands-on 
activities, and demonstrations. The other strategies were not 
considered a part of this group. . Table 3 shows the percent of 

■if 

group (1) activities teachers reported using before the STARS 
program and the percent of group (1) activities after the STARS 
program, A positive or negative difference pre-post is noted. The 
Sign Test shows no significant difference in the amount of group 
(1) activities before and after the STARS program. 

Summary and Conclusions 

1,) Teachers perceive that they received a somewhat substantial 
amount of experience in scientific research design and 
experimentation . 

20 Teachers perceive that they made somewhat substantial gains 
in their acquisition of science content knowledge. In addition, 
teachers perceive that they have incorporated this content 
into their curricula in a somewhat substantial way. 
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3 . ) Teachers perceive that they moderately increased their 
ability to perform science process skills. Pre- and post-testing 
of science process skill attainment shows no significant gains in 
teachers' ability to perform science process skills after 
participation in the STARS program. Possibly, the skills acquired 
by teachers in their internships may be narrow and specific to 
each internship, therefore difficult to measure with a "generic" 
instrument. 

4. ) Teachers perceive that they have somewhat substantially 
increased the incorporation of research/process methodologies 
into their curricula. In addition, teachers perceive that their 
reaching strategies have changed somewhat substantially as a 
result of their summer experiences. 

5. ) Teachers perceive that they have made somewhat substantial 
gains in their understanding and knowledge of applications of 
science in the workplace. 

6. ) Teachers perceive that the learning cycle and inquiry 
strategies learned during the program have been helpful in their 
day-to-day lesson planning. In addition, teachers perceive that 
they have made somewhat substantial gains in their ability to 
design activities for classroom use. 

7. ) Permeating many comments of the teachers on cjuestionnaires 
was the high degree to which they valued the program. The 
teachers are sharing their experiences and perceptions of the 
program with fellow teachers and administrators. 

8. ) The teachers perceive that the program has substantially 
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impacted their ability to implement hands-on laboratory- 
oriented curricula. However, the Science Classroom Activity 
Checklist and the pre-to-post program questionnaire on teaching 
strategies do not reflect an increase in the use of these 
strategies in the classroom. These data suggest that either (a) 
the methods we are using are not detecting changes that exist, or 
(b) that observable, measurable change is not taking place. 
Analysis of the field notes of classroom observations may yield 
data that contribute to the resolution of this dichotomy. 

The data from teacher interviews and questionnaires suggest 
that some type of change is occurring in teacher practice. This 
change may involve the way in which a teacher views what she or 
he does; after the internship experience teachers appear to view 
their roles in their classrooms through different "lenses," which 
impacts their decision making in ways we have yet to measure. 
Researching changes in teacher decision making may be an 
interesting and fruitful area of research regarding teacher 
change as a result of participating in a research internship. 

Our future research projects will focus on gaining 
understanding regarding changes that occur in teachers' 
knowledge, skills, and attitudes? how these changes translate 
into changes in classroom practice, teacher professionalism, and 
teachers' abilities to act as change agents; and the components 
or facets of scientific work experience programs that foster 
those changes. This information will be important to the 
continuing growth of the scientific work experience program as a 
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model of science teacher developxnent in ways that will best 
promote excellence in science teaching across the United States. 
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Table 1 



MEANS OP PRE- AND POST-TEST SCORES OBTAII ON THE 
TEST OF INTEGRATED PROCESS SKILLS (TIPS) 

n=17 

Pre-test Mean Post-test Mean t p>.05 

28 29.8 0.044 ns 

CV=1.753 
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Table 3 

PRE TO POST PROGIWMl QUESTIONNAIRE RESULTS REGARDING 
I*ABORATORY AND/OR HANDS-ON TEACHING STRATEGIES 



% class time spent on 
lab/hands-on activities 

Teacher Pre-proaram Time Post-procfram Time Difference 



Tl 


5% 


25% 


+ 


T2 


20% 


3 0% 


+ 


T3 


30% 


40% 


+ 


T4 


34% 


30% 




T5 


35% 


60% 


+ 


T6 


70% 


65% 




T7 


15% 


25% 


+ 


T8 


46% 


51% 


+ 



Ho= there is no significant difference between the amount of time 
spent on lab/hands~on acitivities before participation in the 
STARS program and after participation in the STARS program. 

n=8 6 positive differences 
2 negative differences 
P(X<2 or X>6) = 0.28908 

Since the probability is greater than 0.05, do not reject 
Ho. 
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